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® A coding system for video signals. 



® A video processing system is disclosed that 
separates and separately encodes and decodes the 
low (LFREQ) and high (HFREQ) spatial frequency 
coefficients of images for transmission or storage. 
Each block of an image is transformed (in 106) into 
the frequency domain. High frequency coefficients of 
the resulting transform matrix are separated (in 108) 
from the low frequency coefficients. The low fre- 
quency coefficients are motion prediction compen- 
sated (by 124) to derive motion vectors and a pre- 



diction error signal. The motion vectors, prediction 
error signal and high frequency coefficients are 
channel encoded (in 116) for storage or transmis- 
sion. In a receiver, the motion vectors and prediction 
error signal are used to reconstruct a low frequency 
motion-compensated version of the Image. The high 
frequency coefficients are inverse transformed into 
the pel domain and are combined with the recon- 
structed low frequency version of the image to re- 
construct a version of the original image. 
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TechnicalFleld 

This invention relates to signal coding systems 
and. more particularly, to encoding and decoding 
video signals of moving images suitable for trans- 
mission or for storage. 



Backgroundof the Invention 



Prior video signal interframe coding, transmis- 
sion and reproduction systems initially encode a 
base frame image. This base frame image incor- 
porates both the low and the high spatial frequency 
components of the image. The base frame image 
IS transmitted and thereafter only information repre- 
sentative of the difference between the base frame 
and each subsequent image is transmitted. Both 
low and high frequency spatial components are 
contained within the difference information. Using 
he base frame image and the subsequent dif- 
ference information a high quality version, i e an 
accurate or error free representation, of any 'de- 
sired subsequent image can be reconstructed If 
dif^ererice information is lost during transmission, 
only a low quality version of the desired image i e 
an inaccurate or error strewn representation, ' can 
be reconstructed. Such prior arrangements are in- 
efficient because they lack the ability to rapidly 
reproduce high quality images in random access 
applications. Similarly, this prior arrangement re- 
quires a long time to recover from loss of the 
difference infomnation caused by transmission er- 
rors. The inefficiency and delays are caused by the 
leveraged encoding of each image beyond the 
base frame since all of the information prior to the 
selected frame is required, i.e.. ail the information 
contained in the base frame and all the difference 
information relating to each sequentially subse- 
quent image. However, it is known that interframe 
coding tends to reduce the number of bits that are 
required to be transmitted; 

In a prior attempt to reduce the reconstruction 
difficulty a new base frame is periodically incor- 
porated mto the bit stream However, this technique 
dramatically increases the average number of bits 
required to represent all the information in an aver- 
age frame because each base frame is encoded 
using interframe coding and all of the information 
representing a base frame is used by each subse- 
quent image. It is known that intraframe coding 
ends to require a larger number of bits than inter- 
frame coding, in intraframe coding, characteristics 
of neighbonng picture elements (pels or pixels) are 
predicted based on the values of those characteris- 
tics of neighboring pels in the same frame, and the 
error or difference between the actual value and 
the predicted value is encoded. This type of en- 
coding is illustrated in an article entitled "Adaptive 



Coding of Monochrome and Color Images"' IEEE 
Trans. Communications. Vol. C0M-2S. po i285- 
1292, Nov. 1977. 

5 Summaryof the Invfentton 

The problems with prior image coding and 
reproduction systems are overcome, in accordance 
with an aspect of the invention, by encoding and 
w transmitting a low spatial frequency representation 
of an image using interframe prediction techniques 
High spatial frequency coefficients of the image are 
encoded directly for transmission. 
. In a specific embodiment of an encoder a 
's video signal is divided into blocks comprising 'an 
array of pels. Each block is transformed Into the 
frequency domain. The high and the low frequency 
coefficients of each block are separately extracted 
from the resulting transfomied matrix. A version of 
20 the original image represented by the low fre- 
quency coefficients of each block is motion com- 
pensated to yield a prediction error signal and 
motion vectors. The prediction error signal the 
motion vectors and the high frequency coefficients 
25 are encoded using a method suitable to the me- 
dium of transmission or storage. The prediction 
error signal and motion vectors are assigned a hiqh 
pnonty. The high frequency coefficients are as- 
signed a low priority and may be dropped by a 
30 network during congestion or not retrieved from 
storage. 

A receiver decodes and separates the predic- 
lon error signal, the motion vectors and the hiqh 
frequency coefficients. The prediction error signal 

35 and motion vectors are used to reconstruct a mo- 
tion compensated version of the low frequency 
image. The high frequency coefficients are inverse 
transformed and the resulting reconstructed high 
frequency version of the image is combined with 

00 the reconstructed, motion compensated low fre- 
quency version of the image to produce a re- 
constructed version of the original image 



45 



BriefPescrlption of the Drawing 



FIG. 1 shows, in simplified block diagram form 
an encoder embodying aspects of the invention' 
and ' 

FIG. 2 shows, in simplified block diagram form 
so a decoder embodying aspects of the invention. ' 



DetalledPescription 



There is some experimental evidence that for 
55 regions of high motion in a sequence of movinq 
image frames the interframe correlation of hiqh 
frequency coefficients of a discrete cosine trans- 
form (OCT) of those regions is approximately z-ro 
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(0). Therefore, the bit-rate to code these coeffi- 
cients in an intraframe format is not much different 
than coding them in an interframe frame or Motion 
Compensated format FIG. 1 shows an encoder that 
encodes the high frequency coefficients of a DCT 
of a sequence of image frames in an intraframe 
format. An additional DCT beyond those used in 
prior motion compensated systems is used as a 
low-pass loop-filter to generate a low frequency 
prediction. 

In an example encoder, an original video signal 
VIDIN is first delayed by delay 102 for reasons to 
be given later. The delayed signal 104 is then 
converted from the pel domain to the Discrete 
Cosine Transform domain by DCT 106. An exam- 
ple DCT 106 groups input pels, i.e. picture ele- 
ments, into 2- dimensional blocks, e.g., 8x8 pels 
each. DCT 106 produces a set of output frequency 
coefficients that are grouped into 2-dimensional 
blocks of the same size as the input blocks. The 
transform coefficients supplied as an output by 
OCT 106 are then partitioned by separator 108 into 
two groups, namely, high spatial frequency coeffi- 
cients and low spatial frequency coefficients. The 
high spatial frequency coefficients. HFREQ. are 
selected based upon a predetermined threshold 
and are supplied to quantizer 110. The remaining 
low spatial frequency coefficients, LFREQ, are sup- 
plied to subtracter 112. 

High spatial frequency coefficients HFREQ are 
quantized by quantizer 110. A quantizer reduces 
the number of levels available for coefficients to 
assume. The quantizing can optionally be made 
responsive to the quantity of information stored in 
elastic store 114 so as to prevent elastic store 114 
from overflowing. The output from quantizer 110. 
QHFREQ, is supplied to elastic store 114. Elastic 
store 114 subsequently supplies QHFREQ to chan- 
nel coder 116 for encoding appropriate for trans- 
mission or storage, depending on the application. If 
elastic store 114 is not present. QHFREQ is sup- 
plied directly to channel coder 116 for encoding. 
Low spatial frequency coefficients LFREQ are cod- 
ed by interframe prediction techniques that are well 
known In the art and exploit the correlation nor- 
mally present between image frames. For example, 
here we show simple Motion Compensation predic- 
tion. (See Digital Pictures Representation and Com- 
pression by Arun N. Netravali and Barry G. Haskell. 
Plenum Press 1988 pp. 334-340.) Other methods 
might include Conditional Motion Compensated In- 
terpolation (see co-pending application Serial Num- 
ber 413.520 Filed on September 27, 1989 and 
allowed on April 5. 1990). 

Low frequency coefficient prediction signal 
PRED. supplied to subtracter 112. is subtracted 
from the low frequency coefficients LFREQ to yield 
an initial low frequency prediction error signal 



LPERR. Low frequency prediction error signal 
LPERR is quantized by quantizer 118 and then 
supplied as low frequency prediction error signal 
QLPERR to elastic store 114. In turn. QLPERR is 
5 subsequently encoded and supplied for transmis- 
sion by channel coder 116. Data supplied to chan- 
nel coder 116 from elastic store 114 is encoded 
using well known reversible data compression 
methods. A serial bit-stream output that can be 
10 transmitted or stored is supplied as an output from 
channel coder 118. 

^ It should be noted that in some applications " 
quantizers 110 and 118 and elastic store 114 may 
be eliminated. If quantizer 118 is eliminated, low 
75 frequency prediction error signal LPERR is sup- 
plied directly to elastic store 114 and adder 120. 
Additionally, a significance detector (not shown) 
may be incorporated into quantizer 118. This de- 
tector would determine whether the values of the 
20 low frequency coefficients in a block of QLPERR 
were too small to warrant transmission, insignificant 
blocks would be replaced by zero (0) before being 
supplied as an output. 

After any significance thresholding is per- 
25 formed, the quantized prediction error signal 
QLPERR is added to low frequency coefficient 
prediction signal PRED by adder 120 to produce 
"reconstructed" low frequency coefficients 
RLFREQ. These reconstructed coefficients are in- 
30 verse transformed by INVERSE DCT 122 to form a 
low frequency pel domain signal LPEL LPEL is 
passed into a frame memory 123 of motion com- 
pensation prediction unit 124. Motion compensation 
prediction units are well known in the art. Original 
35 signal VIDIN is also supplied to motion compensa- 
tion prediction unit 124. Delay 102 is required to 
insure that coefficients LFREQ correspond in time 
to predicted coefficients PRED due to the process- 
ing delay in motion compensation prediction unit 
40 124. Motion compensation prediction unit 124 com- 
pares each block of the original signal VIDIN with 
signal LPEL stored in its frame memory 123 and 
calculates a motion vector for each block. Motion 
vectors are necessary to reconstruct a shift-matrix 
45 for the motion compensated low frequency version 
of the image. The motion vectors are supplied as 
output signal MOVECT to elastic store 114 for 
subsequent transmission by channel coder 116. 
Channel coder 116 assigns a higher priority to the 
so (ow frequency quantized prediction error signal and 
the motion vectors than is assigned to the high 
frequency coefficients. 

Motion compensation prediction unit 124 also 
generates MCPEL. which is a block of pels that is 
55 shifted by an amount corresponding to each of the 
aforementioned motion vectors. MCPEL represents 
the prediction to be used in coding the correspond- 
ing block of original pels now being supplied as an 
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output by delay 102. Thus, an original block of pels 
on signal line 104 passes Into DCT 106 at the 
sanne tinne as the corresponding predicted MCPEL 
block of pels passes into DCT 126. The low fre- 
quency coefficients supplied as an output from 
DCT 126 appear as signal PRED, which is the 
aforementioned prediction for the low frequency 
coefficients LFREQ. DCT 126 does not generate 
any high frequency coefficients. ^ 

A corresponding example decoder is shown in 
FIG. 2. Channel decoder 202 decodes a signal 
supplied from the channel or from storage and 
supplies as outputs signals QHFREQ2, QLPERR2 
and M0VECT2. Under errorless conditions, a sig- 
nal with a suffix of 2 is a reconstructed version of 
the signal having the same base name as shown in 
Fig. 1. High frequency coefficients QHFREQ2 are 
inverse transformed by INVERSE DCT 204 and the 
resulting signal, HPEL. is supplied as an output. 
Signal HPEL represents a high frequency pel do- 
main version of the original image. Quantized low 
frequency coefficient prediction error signal 
QLPERR2 is added to prediction signal PRED2 by 
adder 206 to generate reconstructed low frequency 
coefficients RLFREQ2, which are supplied as an 
output. Reconstructed low frequency coefficients 
RLFREQ2 are inverse transformed by INVERSE 
DCT 208 to form a signal LPEL2 which represents 
a low frequency motion compensated pel domain 
version of the original image. Adder 210 sums 
signals LPEL2 and HPEL and supplies the result as 
output signal VIDOUT. Signal VIDOUT is a full 
frequency reconstructed version of the original vid- 
eo image signal in the pel domain. 

Additionally, low frequency pel signal LPEL2 is 
supplied to a frame memory 21 1 of motion com- 
pensation prediction unit 212. Motion compensation 
prediction unit 212 is also supplied with motion 
vectors M0VECT2. A shifted block of pels, signal 
MCPEL2, which represents an uncorrected motion 
compensation predicted version of the low frequen- 
cy pel domain image is generated by motion com- 
pensation prediction unit 212. Signal MCPEL2 is 
supplied to DCT 214 wherein it is transformed into 
the frequency domain. The resulting output of DCT 
214 is a set of low frequency prediction coeffi- 
cients, PRED2, which are supplied to abovemen- 
tioned adder 206. DCT 214 also does not generate 
any high frequency coefficients. 

At start up, to initialize the motion compensa- 
tion unit and establish a base frame, motion com- 
pensation of the low frequency image is suspended 
for one frame. Thereafter, at predetermined inter- 
vals, motion compensation of the low frequency 
image may be suspended for additional frame 
periods. This is useful in applications requiring 
random access from storage such as compact 
disks or where the Image signal is to be trans- 



ported over a packet network that cannot guaranty 
100% delivery for packets that contain the motion 
vectors and low frequency error signal. Networks 
that guaranty 100% delivery would not require sus- 
5 pension of motion compensation. 

In the encoder (FIG. 1), the time of suspension 
is determined by a control mechanism (not shown) 
and effected by changing the logical condition of 
signal CTRL supplied to motion compensation pre- 
10 diction unit 124. Suspension of motion compensa- 
tion causes signals MOVECT. MCPEL and PRED 
to have values of zero (0). Nothing Is therefore 
subtracted from the low frequency coefficients at 
subtracter 112 and LPERR and low frequency co- 
/5 efficients LFREQ are therefore equal. This causes 
the low frequency coefficients LFREQ to be di- 
rectly encoded rather than prediction error signal 
LPERR and motion vectors MOVECT thereby re- 
sulting in the transmission of a new base frame. 
20 Similarly in the receiver (FIG. 2), a correspond- 

ing time of suspension of motion compensation is 
determined by a control mechanism (also not 
shown) and effected by changing the logical con- 
dition of signal CTRL supplied to motion com- 
25 pensation prediction unit 212. When motion com- 
pensation is suspended in the encoder (FIG. 1) 
signal QLPERR2 in the receiver (FIG. 2) is equal to 
the low frequency coefficients LFREQ (FIG. 1). 
Suspension of motion compensation causes signals 
30 MCPEL2 and PRED2 to have values of zero (0). In 
turn, signal RLFREQ2 is equal to signal QLPERR2 
and correspondingly LPEL2 represents a low fre- 
quency base image. 

Separating the DCT coefficients has several 
35 advantages. First, the low frequency image and the 
high frequency image can be computed in parallel, 
since errors introduced by compressing the low 
frequency information do not translate into addi- 
tional errors in the high frequency Information. 
40 Also, since not all DCT coefficients are necessary 
to generate either the low frequency image or the 
high frequency image, the entire DCT need not be 
computed for each, and an intelligent Implementa- 
tion of the DCT can improve the efficiency of the 
45 computation. Additionally, a better prediction of the 
coefficients in the motion compensation loop is 
obtained, since fewer coefficients are predicted. 
Furthermore, by motion compensating only the low 
frequencies, the overall image quality is not sacri- 
50 ficed in areas with large motion. In this case, mo- 
tion compensation cannot adequately predict the 
high frequencies anyway, so little is lost by not 
motion conripensating them. This coding method 
has all the advantages for packet transmission in- 
55 herent in any variable bit-rate method. A balance is 
struck between data compression and robustness 
. to packet loss, by motion compensating iow fre- 
quencies, yet intraframe coding the high frequen- 
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cies. 

An embedded code, as is well known, is a 
digital code that allows bits or symbols to be 
dropped and inserted without causing mismatch 
between the memories of the encoder and the 
decoder The encoding and decoding system de- 
scribed above is such a system. Thus, another 
advantage of this system is that if some of the high 
frequency coefficients transmitted by the encoder 
are lost they can be zero-filled at the decoder and 
an image with less error than prior systems will be 
displayed. Further, only the low frequency portions 
of the intraframe coded base frame and subse- 
quent interframe coded frames until ad including a 
desired random frame need be read. These are 
then combined with the high frequency portion of 
the desired frame and displayed. Therefore, a high 
quality Image can be randomly accessed more 
rapidly than with prior coded video systems. 

Claims 

1. Apparatus for use in a video coding system, 
CHARACTERISED BY means for receiving as 
an input an original digitized video image sig- 
nal, means (104) for delaying said video image 
signal, means (106) for deriving frequency 
transform coefficients from said delayed video 
image signal, means (108) for separating said 
frequency transform coefficients into a set of 
low frequency coefficients and a set of high 
frequency coefficients, said separation being 
based upon a predetermined threshold, means 
(112.118.120.122.124.126) responsive to said 
set of low frequency coefficients and said Input 
video image signal for generating and sup- 
plying as an output a representation of a mo- 
tion compensated low frequency version of 
said original video image signal that includes a 
set of motion vectors and a prediction error 
signal, means (116) for encoding said set of 
high frequency coefficients, said prediction er- 
ror signal and said set of motion vectors into a 
predetermined format specific to a predeter- 
mined output medium, and means for interfac- 
ing said predetermined format to said pre- 
determined output medium. 

2. Apparatus as claimed in claim 1 including elas- 
tic storage means (114) for storing said set of 
high frequency coefficients, said prediction er- 
ror signal and said set of motion vectors and 
for supplying said set of high frequency coeffi- 
cients, said prediction error signal and said set 
of motion vectors as an output to said means 
for encoding. 

3. Apparatus as claimed in claim 1 or 2 including 



means (110) for quantizing said set of high 
frequency coefficients. 

4. Apparatus as claimed in claim 1, 2 or 3 includ- 
5 ing means (112) for generating an Initial error 

signal, and means (118) for quantizing respon- 
sive to said initial error signal and supplying as 
an output said prediction error signal. 

70 5. Apparatus as claimed in claim 1, 2 or 3 
wherein said means for generating and sup- 
plying includes means (122) for generating a 
reconstructed low frequency pel domain ver- 
sion of said digitized video image signal, mo-- 
15 tion compensation prediction means (123,124) 

responsive to a current frame of said original 
digitized video image signal and an at least 
one stored frame of said reconstructed low 
frequency pel domain version of said digitized 
20 video image signal for. generating a motion 

compensated pel domain prediction signal and 
motion vectors, means (126) for transforming 
said motion compensated pel domain predic- 
tion signal into said set of low frequency pre- 
ss diction coefficients, and means (112) respon- 
sive to said set of low frequency coefficients 
and said set of low frequency prediction coeffi- 
cients for generating and supplying said pre- 
diction error signal. 

30 

6. Apparatus as claimed in claim 5 wherein said 
means for generating and supplying includes 
mean (120) for supplying as an output a signal 
representing a combination of said low fre- 

35 quency prediction coefficients and said predic- 

tion error signal to said means (122) for gen- 
erating said reconstructed low frequency pel 
domain version of said digitized video image 
signal. 

40 

7, Apparatus as claimed in claim 5 wherein said 
means for generating and supplying includes 
means (118) for supplying a quantized version 
of said frequency domain prediction error sig- 

45 nal, and means (120) for supplying as an out- 

put a signal representing the combination of 
said low frequency prediction coefficients and 
quantized version of said frequency domain 
prediction error signal to said means (122) for 

50 generating said reconstructed low frequency 

pel domain version of said digitized video im- 
age signal. 

8- Apparatus for use in a video decoding system. 
55 CHARACTERISED BY means (202) fcr inter- 

facing to a predetermined Input medium for 
receiving, separating, decoding and sjpplying 
as an output a set of high frequency coeffi- 
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cients, a prediction error signal and a set of 
motion vectors from data supplied from said 
medium in a predetermined format, means 
(204) for transforming said set of high fre- 
quency coefficients into a high frequency pel 
domain version of an original video image 
means (206,208.212,214) responsive to said 
prediction error signal and said set of motion 
vectors for deriving a motion compensated low 
frequency pel domain version of said original 
video image signal, means (210) for combining 
said high frequency pel domain version of said 
original video image and said motion com- 
pensated low frequency pel domain version of 
said onginal video image signal into a recon- 
structed full frequency version of said original 
video image signal in the pel domain, and . 
means for supplying said reconstructed full 
frequency version of said original video image 
signal in the pel domain as an output. 

9. Apparatus as claimed in claim 8 wherein said 
means for deriving includes motion compensa- 
tion prediction means (212) responsive to said 
motion vectors and ah at least one prior stored 
frame of a reconstructed low frequency pel 
domain version of said digitized video image 
signal for generating a current motion com- 
pensated prediction pel domain signal, means 
(214) for transforming said current motion 
compensated pel domain prediction signal into 
a set of low frequency prediction coefficients 
means (206) responsive to said set of low 
frequency prediction coefficients and said fre- 
quency domain error, signal for generating a 
set of reconstructed low frequency prediction 
coefficients, means (208) supplied with said 
reconstructed low frequency prediction coeffi- 
cients for generating a reconstructed low fre- 
quency pel domain version of said digitized < 
video image signal, and means (21 1) for stor- 
ing said low frequency pel domain version of 
said digitized video image signal. 

10. Apparatus as claimed in claim 8 or 9 wherein 4 
said means for receiving, separating, decoding 
and supplying further includes means for sup- 
plying null value high frequency coefficients if 
said high frequency coefficients are not sup- 
phed from said data within a predetermined sc 
length of time. 
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